Systems and methods for transitioning between modes of tracking real-world objects for artificial reality interfaces

ABSTRACT

The disclosed computer-implemented method may include tracking (1) a position of a primary real-world object within a real-world environment via a primary tracking method, and (2) a position of a secondary real-world object within the real-world environment via a secondary tracking method. The method may further include presenting (1) a primary virtual object at a position within an artificial environment corresponding to the tracked position of the primary real-world object, and (2) a secondary virtual object at a position within the artificial environment corresponding to the tracked position of the secondary real-world object. The method may further include (1) detecting an interaction of the primary real-world object with the secondary real-world object, and (2) transitioning to tracking the position of the primary real-world object via the secondary tracking method. Various other methods, systems, and computer-readable media are also disclosed.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/769,542, filed Nov. 19, 2018, the disclosure of which isincorporated, in its entirety, by this reference.

BACKGROUND

Artificial reality systems, such as virtual reality (VR) systems and/oraugmented reality (AR) systems, may provide thrilling experiences thatmay be more immersive than almost any other digital entertainment orsimulation experience available today. Artificial reality systems mayenable users to travel through space and time, interact with friends inthree-dimensional worlds, or play video games in radically redefinedways. Artificial reality systems may also be used for purposes otherthan recreation. Governments may use them for military trainingsimulations, doctors may use them to practice surgery, and engineers mayuse them as visualization aids. Artificial reality systems may also beused for productivity purposes. Information organization, collaboration,and privacy may all be enabled or enhanced through the use of artificialreality systems.

Unfortunately, it may be difficult for users to interact with real-worldobjects (e.g., controllers, keyboards, writing tools, furniture, etc.)while interacting with some artificial reality systems. For example,some artificial reality systems may include head-worn display systemsand/or near-eye displays (NEDs) that, when worn by a user, may obstructa line of sight between the user's eyes and one or more real-worldobjects. This lack of visual feedback may cause inefficiencies in userinteraction with real-world objects while users are wearing suchdevices. This may be particularly problematic when such real-worldobjects may include the user's hands and/or one or more input devicesassociated with the artificial reality system (e.g., an artificialreality input device, a game controller, etc.). For example, it may bedifficult for a user to locate and/or pick up a game controllerassociated with the artificial reality system while wearing a head-worndisplay system.

Furthermore, modern artificial reality systems may include and/or may beassociated with various systems for tracking physical objects within areal-world environment. Some of these tracking systems may employtracking methods that may have comparative differences (e.g., accuracy,resolution, power efficiency, etc.) that may make one tracking methodmore suitable for a particular situation than another tracking method.Unfortunately, conventional artificial reality systems may be unable toeffectively and/or efficiently transition between tracking methods whenit may be advantageous to do so.

Hence, the instant disclosure identifies and addresses a need for newsystems and methods for transitioning between modes of trackingreal-world objects for artificial reality interfaces.

SUMMARY

As will be described in greater detail below, the instant disclosuredescribes various systems and methods for transitioning between modes oftracking real-world objects for artificial reality interfaces.Embodiments of the systems and methods described herein may trackpositions of a primary and a secondary real-world object within areal-world environment via a respective primary and secondary trackingmethod (e.g., a computer vision tracking method, an optical trackingmethod, an inertial tracking method, etc.). Embodiments may also presenta primary and a secondary virtual object within an artificialenvironment at respective positions corresponding to the positions ofthe primary and secondary objects within the real-world environment.Embodiments may also detect an interaction of the primary real-worldobject with the secondary real-world object, and may transition fromtracking the position of the primary real-world object within thereal-world environment via the primary tracking method to tracking theposition of the primary real-world object within the real-worldenvironment via the secondary tracking method in response to detectingthe interaction of the primary real-world object with the secondaryreal-world object.

In one example, a computer-implemented method for transitioning betweenmodes of tracking real-world objects for artificial reality interfacesmay include tracking (1) a position of a primary real-world objectwithin a real-world environment via a primary tracking method, and (2) aposition of a secondary real-world object within the real-worldenvironment via a secondary tracking method. The computer-implementedmethod may further include presenting (1) a primary virtual object thatrepresents the primary real-world object at a position within anartificial environment corresponding to the position of the primaryreal-world object within the real-world environment, and (2) a secondaryvirtual object that represents the secondary real-world object at aposition within the artificial environment corresponding to the positionof the secondary real-world object within the real-world environment.The computer-implemented method may further include detecting aninteraction of the primary real-world object with the secondaryreal-world object, and transitioning from tracking the position of theprimary real-world object within the real-world environment via theprimary tracking method to tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method in response to detecting the interaction of the primaryreal-world object with the secondary real-world object.

In at least one embodiment, the method may further include (1) detectingan additional interaction of the primary real-world object with thesecondary real-world object, and (2) transitioning from tracking theposition of the primary real-world object within the real-worldenvironment via the secondary tracking method to tracking the positionof the primary real-world object within the real-world environment viathe primary tracking method in response to detecting the additionalinteraction of the primary real-world object with the secondaryreal-world object.

In one or more embodiments, the primary tracking method may include acomputer vision tracking method and the secondary tracking method mayinclude at least one of: (1) an optical tracking method, (2) asimultaneous localization and mapping (SLAM) tracking method, or (3) aninertial tracking method. In at least one embodiment, the method mayfurther include adjusting, in response to transitioning from trackingthe position of the primary real-world object within the real-worldenvironment via the primary tracking method to tracking the position ofthe primary real-world object within the real-world environment via thesecondary tracking method, an appearance of at least one of (1) theprimary virtual object, or (2) the secondary virtual object.

In some examples, the computer-implemented method may further includedetermining a proximity of the primary real-world object to thesecondary real-world object. In at least one example, detecting theinteraction of the primary real-world object with the secondaryreal-world object may include determining that the proximity of theprimary real-world object to the secondary real-world object is lessthan a predetermined threshold. In one or more examples, thecomputer-implemented method may further include adjusting, based on theproximity of the primary real-world object to the secondary real-worldobject, an appearance of at least one of (1) the primary virtual object,or (2) the secondary virtual object.

In some embodiments, the computer-implemented method may further includepresenting, in response to transitioning from tracking the position ofthe primary real-world object within the real-world environment via theprimary tracking method to tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method, at the position within the artificial environmentcorresponding to the position of the primary real-world object withinthe real-world environment, a unified virtual object that representsboth the primary virtual object and the secondary virtual object.

In at least one embodiment, the primary real-world object may include ahand of a user, and the secondary real-world object may include anartificial reality controller device. In one or more embodiments, thesecondary real-world object may include a touch sensor, and detectingthe interaction of the primary real-world object with the secondaryreal-world object may include detecting, via the touch sensor, a touchof the secondary real-world object by the primary real-world object.

In some examples, the computer-implemented method may further includedetermining that the primary real-world object may include one of (1) aleft hand of a user, or (2) a right hand of the user. In at least oneexample, the secondary real-world object may include an artificialreality controller device configured to be operated by the user via oneof (1) the left hand of the user, or (2) the right hand of the user. Inone or more examples, the computer-implemented method may furtherinclude presenting a notification to the user upon detecting theinteraction of the primary real-world object with the secondaryreal-world object and when at least one of (1) the artificial realitycontroller device is configured to be operated by the right hand of theuser and upon determining that the primary real-world object comprisesthe left hand of the user, or (2) the artificial reality controllerdevice is configured to be operated by the left hand of the user andupon determining that the primary real-world object comprises the righthand of the user.

In addition, a corresponding system for transitioning between modes oftracking real-world objects for artificial reality interfaces mayinclude several modules stored in memory, including a tracking modulethat tracks (1) a position of a primary real-world object within areal-world environment via a primary tracking method, and (2) a positionof a secondary real-world object within the real-world environment via asecondary tracking method. The system may also include a presentingmodule that presents (1) a primary virtual object that represents theprimary real-world object at a position within an artificial environmentcorresponding to the position of the primary real-world object withinthe real-world environment, and (2) a secondary virtual object thatrepresents the secondary real-world object at a position within theartificial environment corresponding to the position of the secondaryreal-world object within the real-world environment.

The system may further include a detecting module that detects aninteraction of the primary real-world object with the secondaryreal-world object. The system may also include a transitioning modulethat transitions from tracking the position of the primary real-worldobject within the real-world environment via the primary tracking methodto tracking the position of the primary real-world object within thereal-world environment via the secondary tracking method in response todetecting the interaction of the primary real-world object with thesecondary real-world object. The system may also include at least onephysical processor that executes the tracking module, the presentingmodule, the detecting module, and the transitioning module.

In at least one embodiment, the detecting module may further detect anadditional interaction of the primary real-world object with thesecondary real-world object, and the transitioning module may furthertransition from tracking the position of the primary real-world objectwithin the real-world environment via the secondary tracking method totracking the position of the primary real-world object within thereal-world environment via the primary tracking method in response todetecting the additional interaction of the primary real-world objectwith the secondary real-world object.

In one or more embodiments, the presenting module may further adjust, inresponse to the transitioning module transitioning from tracking theposition of the primary real-world object within the real-worldenvironment via the primary tracking method to tracking the position ofthe primary real-world object within the real-world environment via thesecondary tracking method, an appearance of at least one of (1) theprimary virtual object, or (2) the secondary virtual object.

In some examples, the detecting module may further determine a proximityof the primary real-world object to the secondary real-world object. Inat least one example, the detecting module may detect the interaction ofthe primary real-world object with the secondary real-world object bydetermining that the proximity of the primary real-world object to thesecondary real-world object is less than a predetermined threshold. Inone or more examples, the presenting module may adjust, based on theproximity of the primary real-world object to the secondary real-worldobject, an appearance of at least one of (1) the primary virtual object,or (2) the secondary virtual object.

In some embodiments, the secondary real-world object may include anartificial reality controller device configured to be operated by a uservia one of (1) a left hand of the user, or (2) a right hand of the user.In at least one embodiment, the detecting module further determines thatthe primary real-world object may include one of (1) the left hand ofthe user or (2) the right hand of the user. In one or more embodiments,the presenting module may further present a notification to the userupon the detecting module detecting the interaction of the primaryreal-world object with the secondary real-world object and when at leastone of (1) the artificial reality controller device is configured to beoperated by the right hand of the user and upon the detecting moduledetermining that the primary real-world object comprises the left handof the user, or (2) the artificial reality controller device isconfigured to be operated by the left hand of the user and upon thedetecting module determining that the primary real-world objectcomprises the right hand of the user.

In some examples, the above-described method may be encoded ascomputer-readable instructions on a computer-readable medium. Forexample, a computer-readable medium may include one or morecomputer-executable instructions that, when executed by at least oneprocessor of a computing device, may cause the computing device to track(1) a position of a primary real-world object within a real-worldenvironment via a primary tracking method, an (2) a position of asecondary real-world object within the real-world environment via asecondary tracking method. The computer-readable medium may furtherinclude one or more computer-executable instructions that, when executedby the processor of the computing device, may cause the computing deviceto present (1)+a primary virtual object that represents the primaryreal-world object at a position within an artificial environmentcorresponding to the position of the primary real-world object withinthe real-world environment, and (2) a secondary virtual object thatrepresents the secondary real-world object at a position within theartificial environment corresponding to the position of the secondaryreal-world object within the real-world environment.

The computer-readable medium may further include one or morecomputer-executable instructions that, when executed by the processor ofthe computing device, may cause the computing device to detect aninteraction of the primary real-world object with the secondaryreal-world object. The computer-readable medium may further include oneor more computer-executable instructions that, when executed by theprocessor of the computing device, may cause the computing device totransition from tracking the position of the primary real-world objectwithin the real-world environment via the primary tracking method totracking the position of the primary real-world object within thereal-world environment via the secondary tracking method in response todetecting the interaction of the primary real-world object with thesecondary real-world object.

Features from any of the above-mentioned embodiments may be used incombination with one another in accordance with the general principlesdescribed herein. These and other embodiments, features, and advantageswill be more fully understood upon reading the following detaileddescription in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate a number of exemplary embodimentsand are a part of the specification. Together with the followingdescription, these drawings demonstrate and explain various principlesof the instant disclosure.

FIG. 1-3 illustrate various artificial reality systems that mayimplement, include and/or be included in one or more systems fortransitioning between modes of tracking real-world objects forartificial reality interfaces.

FIG. 4 is a block diagram of an example system for transitioning betweenmodes of tracking real-world objects for artificial reality interfaces.

FIG. 5 is a block diagram of an example implementation of an examplesystem for transitioning between modes of tracking real-world objectsfor artificial reality interfaces.

FIG. 6 is a flow diagram of an example method for transitioning betweenmodes of tracking real-world objects for artificial reality interfaces.

FIG. 7 is a perspective view of an example hand-held controller for anartificial reality system.

FIG. 8 is a view of an example implementation of an example system fortransitioning between modes of tracking real-world objects forartificial reality interfaces.

FIGS. 9-13 show various views of virtual objects within an artificialenvironment in accordance with one or more embodiments of the systemsand methods disclosed herein.

Throughout the drawings, identical reference characters and descriptionsindicate similar, but not necessarily identical, elements. While theexemplary embodiments described herein are susceptible to variousmodifications and alternative forms, specific embodiments have beenshown by way of example in the drawings and will be described in detailherein. However, the exemplary embodiments described herein are notintended to be limited to the particular forms disclosed. Rather, theinstant disclosure covers all modifications, equivalents, andalternatives falling within the scope of the appended claims.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present disclosure is generally directed to systems and methods fortransitioning between modes of tracking real-world objects forartificial reality interfaces. As will be explained in greater detailbelow, embodiments of the instant disclosure may track a position of aprimary real-world object (e.g., a user's hand, an input device, a gamecontroller, etc.) within a real-world environment via a primary trackingmethod (e.g., a computer vision tracking method, an optical trackingmethod, an inertial tracking method, etc.). Embodiments of thisdisclosure may also track a position of a secondary real-world object(e.g., another hand of the user, a hand of another user, another inputdevice, another game controller, etc.) within the real-world environmentvia a secondary tracking method (e.g., another computer vision trackingmethod, another optical tracking method, another inertial trackingmethod, etc.).

Additionally, embodiments of the instant disclosure may present aprimary virtual object (e.g., a computer-generated model, a graphicalobject, etc.) that represents the primary real-world object at aposition within an artificial environment corresponding to the positionof the primary real-world object within the real-world environment.Likewise, an embodiment may also present a secondary virtual object(e.g., another computer-generated model, another graphical object, etc.)that represents the secondary real-world object at a position within theartificial environment corresponding to the position of the secondaryreal-world object within the real-world environment.

An embodiment may also detect an interaction (e.g., a touch, a contact,an approach within a predetermined distance, etc.) of the primaryreal-world object with the secondary real-world object, such as a userpicking up an artificial reality controller device with one of his orher hands. In response to detecting the interaction, an embodiment maytransition from tracking the position of the primary real-world objectwithin the real-world environment via the primary tracking method totracking the position of the primary real-world object within thereal-world environment via the secondary tracking method.

As will be explained in further detail below, the systems and methodsdescribed herein may improve an ability of the user to interact withreal-world objects while using an artificial reality system (e.g., a VRsystem, an AR system, a combination thereof, and so forth). For example,as a user interacts with an artificial reality system that includes, isin communication with, and/or is associated with an embodiment of thesystems and methods described herein, the user may be able to visualizeinteractions between real-world objects as they occur, such as the userreaching his or her hand toward an artificial reality controller deviceand/or the user picking up the artificial reality controller device.Furthermore, by facilitating smooth transitions between methods fortracking real-world objects, the systems and methods described hereinmay enable artificial reality systems to more efficiently and/oreffectively utilize various available tracking methods to trackreal-world objects, thereby improving various aspects of artificialreality systems such as power usage characteristics, situational and/orabsolute tracking accuracy, interactivity with real and/or virtualobjects and/or environments, and so forth.

The following will provide, with reference to FIGS. 1-3, detaileddescriptions of various artificial reality systems. Additionally, thefollowing will provide, with reference to FIGS. 4-5 and FIGS. 6-13,detailed descriptions of systems for transitioning between modes oftracking real-world objects for artificial reality interfaces. Detaileddescriptions of corresponding computer-implemented methods will also beprovided in connection with FIG. 6.

Embodiments of the instant disclosure may include or be implemented inconjunction with various types of artificial reality systems. Artificialreality is a form of reality that has been adjusted in some mannerbefore presentation to a user, which may include, e.g., a virtualreality (VR), an augmented reality (AR), a mixed reality (MR), a hybridreality, or some combination and/or derivative thereof. Artificialreality content may include completely generated content or generatedcontent combined with captured (e.g., real-world) content. Theartificial reality content may include video, audio, haptic feedback, orsome combination thereof, any of which may be presented in a singlechannel or in multiple channels (such as stereo video that produces athree-dimensional effect to the viewer). Additionally, in someembodiments, artificial reality may also be associated withapplications, products, accessories, services, or some combinationthereof, that are used to, e.g., create content in an artificial realityand/or are otherwise used in (e.g., to perform activities in) anartificial reality.

Artificial reality systems may be implemented in a variety of differentform factors and configurations. Some artificial reality systems may bedesigned to work without near-eye displays (NEDs), an example of whichis AR system 100 in FIG. 1. Other artificial reality systems may includean NED that also provides visibility into the real world (e.g., ARsystem 200 in FIG. 2) or that visually immerses a user in an artificialreality (e.g., VR system 300 in FIG. 3). While some artificial realitydevices may be self-contained systems, other artificial reality devicesmay communicate and/or coordinate with external devices to provide anartificial reality experience to a user. Examples of such externaldevices include hand-held controllers, mobile devices, desktopcomputers, devices worn by a user, devices worn by one or more otherusers, and/or any other suitable external system.

Turning to FIG. 1, AR system 100 generally represents a wearable devicedimensioned to fit about a body part (e.g., a head) of a user. As shownin FIG. 1, AR system 100 may include a frame 102 and a camera assembly104 that is coupled to frame 102 and configured to gather informationabout a local environment by observing the local environment. AR system100 may also include one or more audio devices, such as output audiotransducers 108(A) and 108(B) and input audio transducers 110. Outputaudio transducers 108(A) and 108(B) may provide audio feedback and/orcontent to a user, and input audio transducers 110 may capture audio ina user's environment.

As shown, AR system 100 may not necessarily include an NED positioned infront of a user's eyes. AR systems without NEDs may take a variety offorms, such as head bands, hats, hair bands, belts, watches, wristbands, ankle bands, rings, neckbands, necklaces, chest bands, eyewearframes, and/or any other suitable type or form of apparatus. While ARsystem 100 may not include an NED, AR system 100 may include other typesof screens or visual feedback devices (e.g., a display screen integratedinto a side of frame 102).

The embodiments discussed in this disclosure may also be implemented inAR systems that include one or more NEDs. For example, as shown in FIG.2, AR system 200 may include an eyewear device 202 with a frame 210configured to hold a left display device 215(A) and a right displaydevice 215(B) in front of a user's eyes. Display devices 215(A) and215(B) may act together or independently to present an image or seriesof images to a user. While AR system 200 includes two displays,embodiments of this disclosure may be implemented in AR systems with asingle NED or more than two NEDs.

In some embodiments, AR system 200 may include one or more sensors, suchas sensor 240. Sensor 240 may generate measurement signals in responseto motion of AR system 200 and may be located on substantially anyportion of frame 210. Sensor 240 may include a position sensor, aninertial measurement unit (IMU), a depth camera assembly, or anycombination thereof. In some embodiments, AR system 200 may or may notinclude sensor 240 or may include more than one sensor. In embodimentsin which sensor 240 includes an IMU, the IMU may generate calibrationdata based on measurement signals from sensor 240. Examples of sensor240 may include, without limitation, accelerometers, gyroscopes,magnetometers, other suitable types of sensors that detect motion,sensors used for error correction of the IMU, or some combinationthereof.

AR system 200 may also include a microphone array with a plurality ofacoustic sensors 220(A)-220(J), referred to collectively as acousticsensors 220. Acoustic sensors 220 may be transducers that detect airpressure variations induced by sound waves. Each acoustic sensor 220 maybe configured to detect sound and convert the detected sound into anelectronic format (e.g., an analog or digital format). The microphonearray in FIG. 2 may include, for example, ten acoustic sensors: 220(A)and 220(B), which may be designed to be placed inside a correspondingear of the user, acoustic sensors 220(C), 220(D), 220(E), 220(F),220(G), and 220(H), which may be positioned at various locations onframe 210, and/or acoustic sensors 220(1) and 220(J), which may bepositioned on a corresponding neckband 205.

The configuration of acoustic sensors 220 of the microphone array mayvary. While AR system 200 is shown in FIG. 2 as having ten acousticsensors 220, the number of acoustic sensors 220 may be greater or lessthan ten. In some embodiments, using higher numbers of acoustic sensors220 may increase the amount of audio information collected and/or thesensitivity and accuracy of the audio information. In contrast, using alower number of acoustic sensors 220 may decrease the computing powerrequired by the controller 250 to process the collected audioinformation. In addition, the position of each acoustic sensor 220 ofthe microphone array may vary. For example, the position of an acousticsensor 220 may include a defined position on the user, a definedcoordinate on the frame 210, an orientation associated with eachacoustic sensor, or some combination thereof.

Acoustic sensors 220(A) and 220(B) may be positioned on different partsof the user's ear, such as behind the pinna or within the auricle orfossa. Or, there may be additional acoustic sensors on or surroundingthe ear in addition to acoustic sensors 220 inside the ear canal. Havingan acoustic sensor positioned next to an ear canal of a user may enablethe microphone array to collect information on how sounds arrive at theear canal. By positioning at least two of acoustic sensors 220 on eitherside of a user's head (e.g., as binaural microphones), AR device 200 maysimulate binaural hearing and capture a 3D stereo sound field aroundabout a user's head. In some embodiments, the acoustic sensors 220(A)and 220(B) may be connected to the AR system 200 via a wired connection,and in other embodiments, the acoustic sensors 220(A) and 220(B) may beconnected to the AR system 200 via a wireless connection (e.g., aBluetooth connection). In still other embodiments, the acoustic sensors220(A) and 220(B) may not be used at all in conjunction with the ARsystem 200.

Acoustic sensors 220 on frame 210 may be positioned along the length ofthe temples, across the bridge, above or below display devices 215(A)and 215(B), or some combination thereof. Acoustic sensors 220 may beoriented such that the microphone array is able to detect sounds in awide range of directions surrounding the user wearing the AR system 200.In some embodiments, an optimization process may be performed duringmanufacturing of AR system 200 to determine relative positioning of eachacoustic sensor 220 in the microphone array.

AR system 200 may further include or be connected to an external device.(e.g., a paired device), such as neckband 205. As shown, neckband 205may be coupled to eyewear device 202 via one or more connectors 230. Theconnectors 230 may be wired or wireless connectors and may includeelectrical and/or non-electrical (e.g., structural) components. In somecases, the eyewear device 202 and the neckband 205 may operateindependently without any wired or wireless connection between them.While FIG. 2 illustrates the components of eyewear device 202 andneckband 205 in example locations on eyewear device 202 and neckband205, the components may be located elsewhere and/or distributeddifferently on eyewear device 202 and/or neckband 205. In someembodiments, the components of the eyewear device 202 and neckband 205may be located on one or more additional peripheral devices paired witheyewear device 202, neckband 205, or some combination thereof.Furthermore, neckband 205 generally represents any type or form ofpaired device. Thus, the following discussion of neckband 205 may alsoapply to various other paired devices, such as smart watches, smartphones, wrist bands, other wearable devices, hand-held controllers,tablet computers, laptop computers, etc.

Pairing external devices, such as neckband 205, with AR eyewear devicesmay enable the eyewear devices to achieve the form factor of a pair ofglasses while still providing sufficient battery and computation powerfor expanded capabilities. Some or all of the battery power,computational resources, and/or additional features of AR system 200 maybe provided by a paired device or shared between a paired device and aneyewear device, thus reducing the weight, heat profile, and form factorof the eyewear device overall while still retaining desiredfunctionality. For example, neckband 205 may allow components that wouldotherwise be included on an eyewear device to be included in neckband205 since users may tolerate a heavier weight load on their shouldersthan they would tolerate on their heads. Neckband 205 may also have alarger surface area over which to diffuse and disperse heat to theambient environment. Thus, neckband 205 may allow for greater batteryand computation capacity than might otherwise have been possible on astand-alone eyewear device. Since weight carried in neckband 205 may beless invasive to a user than weight carried in eyewear device 202, auser may tolerate wearing a lighter eyewear device and carrying orwearing the paired device for greater lengths of time than the userwould tolerate wearing a heavy standalone eyewear device, therebyenabling an artificial reality environment to be incorporated more fullyinto a user's day-to-day activities.

Neckband 205 may be communicatively coupled with eyewear device 202and/or to other devices. The other devices may provide certain functions(e.g., tracking, localizing, depth mapping, processing, storage, etc.)to the AR system 200. In the embodiment of FIG. 2, neckband 205 mayinclude two acoustic sensors (e.g., 220(1) and 220(J)) that are part ofthe microphone array (or potentially form their own microphonesubarray). Neckband 205 may also include a controller 225 and a powersource 235.

Acoustic sensors 220(1) and 220(J) of neckband 205 may be configured todetect sound and convert the detected sound into an electronic format(analog or digital). In the embodiment of FIG. 2, acoustic sensors220(1) and 220(J) may be positioned on neckband 205, thereby increasingthe distance between the neckband acoustic sensors 220(1) and 220(J) andother acoustic sensors 220 positioned on eyewear device 202. In somecases, increasing the distance between acoustic sensors 220 of themicrophone array may improve the accuracy of beamforming performed viathe microphone array. For example, if a sound is detected by acousticsensors 220(C) and 220(D) and the distance between acoustic sensors220(C) and 220(D) is greater than, e.g., the distance between acousticsensors 220(D) and 220(E), the determined source location of thedetected sound may be more accurate than if the sound had been detectedby acoustic sensors 220(D) and 220(E).

Controller 225 of neckband 205 may process information generated by thesensors on neckband 205 and/or AR system 200. For example, controller225 may process information from the microphone array that describessounds detected by the microphone array. For each detected sound,controller 225 may perform a direction of arrival (DoA) estimation toestimate a direction from which the detected sound arrived at themicrophone array. As the microphone array detects sounds, controller 225may populate an audio data set with the information. In embodiments inwhich AR system 200 includes an inertial measurement unit, controller225 may compute all inertial and spatial calculations from the IMUlocated on eyewear device 202. Connector 230 may convey informationbetween AR system 200 and neckband 205 and between AR system 200 andcontroller 225. The information may be in the form of optical data,electrical data, wireless data, or any other transmittable data form.Moving the processing of information generated by AR system 200 toneckband 205 may reduce weight and heat in eyewear device 202, making itmore comfortable to the user.

Power source 235 in neckband 205 may provide power to eyewear device 202and/or to neckband 205. Power source 235 may include, withoutlimitation, lithium ion batteries, lithium-polymer batteries, primarylithium batteries, alkaline batteries, or any other form of powerstorage. In some cases, power source 235 may be a wired power source.Including power source 235 on neckband 205 instead of on eyewear device202 may help better distribute the weight and heat generated by powersource 235.

As noted, some artificial reality systems may, instead of blending anartificial reality with actual reality, substantially replace one ormore of a user's sensory perceptions of the real world with a virtualexperience. One example of this type of system is a head-worn displaysystem, such as VR system 300 in FIG. 3, that mostly or completelycovers a user's field of view. VR system 300 may include a front rigidbody 302 and a band 304 shaped to fit around a user's head. VR system300 may also include output audio transducers 306(A) and 306(B).Furthermore, while not shown in FIG. 3, front rigid body 302 may includeone or more electronic elements, including one or more electronicdisplays, one or more inertial measurement units (IMUS), one or moretracking emitters or detectors, and/or any other suitable device orsystem for creating an artificial reality experience.

Artificial reality systems may include a variety of types of visualfeedback mechanisms. For example, display devices in AR system 200and/or VR system 300 may include one or more liquid crystal displays(LCDs), light emitting diode (LED) displays, organic LED (OLED)displays, and/or any other suitable type of display screen. Artificialreality systems may include a single display screen for both eyes or mayprovide a display screen for each eye, which may allow for additionalflexibility for varifocal adjustments or for correcting a user'srefractive error. Some artificial reality systems may also includeoptical subsystems having one or more lenses (e.g., conventional concaveor convex lenses, Fresnel lenses, adjustable liquid lenses, etc.)through which a user may view a display screen.

In addition to or instead of using display screens, some artificialreality systems may include one or more projection systems. For example,display devices in AR system 200 and/or VR system 300 may includemicro-LED projectors that project light (using, e.g., a waveguide) intodisplay devices, such as clear combiner lenses that allow ambient lightto pass through. The display devices may refract the projected lighttoward a user's pupil and may enable a user to simultaneously view bothartificial reality content and the real world. Artificial realitysystems may also be configured with any other suitable type or form ofimage projection system.

Artificial reality systems may also include various types of computervision components and subsystems. For example, AR system 100, AR system200, and/or VR system 300 may include one or more optical sensors suchas two-dimensional (2D) or three-dimensional (3D) cameras,time-of-flight depth sensors, single-beam or sweeping laserrangefinders, 3D LiDAR sensors, and/or any other suitable type or formof optical sensor. An artificial reality system may process data fromone or more of these sensors to identify a location of a user, to mapthe real world, to provide a user with context about real-worldsurroundings, to track one or more real-world objects, and/or to performa variety of other functions.

Artificial reality systems may also include one or more input and/oroutput audio transducers. In the examples shown in FIGS. 1 and 3, outputaudio transducers 108(A), 108(B), 306(A), and 306(B) may include voicecoil speakers, ribbon speakers, electrostatic speakers, piezoelectricspeakers, bone conduction transducers, cartilage conduction transducers,and/or any other suitable type or form of audio transducer. Similarly,input audio transducers 110 may include condenser microphones, dynamicmicrophones, ribbon microphones, and/or any other type or form of inputtransducer. In some embodiments, a single transducer may be used forboth audio input and audio output.

While not shown in FIGS. 1-3, artificial reality systems may includetactile (i.e., haptic) feedback systems, which may be incorporated intoheadwear, gloves, body suits, hand-held controllers, environmentaldevices (e.g., chairs, floormats, etc.), and/or any other type of deviceor system. Haptic feedback systems may provide various types ofcutaneous feedback, including vibration, force, traction, texture,and/or temperature. Haptic feedback systems may also provide varioustypes of kinesthetic feedback, such as motion and compliance. Hapticfeedback may be implemented using motors, piezoelectric actuators,fluidic systems, and/or a variety of other types of feedback mechanisms.Haptic feedback systems may be implemented independent of otherartificial reality devices, within other artificial reality devices,and/or in conjunction with other artificial reality devices.

By providing haptic sensations, audible content, and/or visual content,artificial reality systems may create an entire virtual experience orenhance a user's real-world experience in a variety of contexts andenvironments. For instance, artificial reality systems may assist orextend a user's perception, memory, or cognition within a particularenvironment. Some systems may enhance a user's interactions with otherpeople in the real world or may enable more immersive interactions withother people in a virtual world. Artificial reality systems may also beused for educational purposes (e.g., for teaching or training inschools, hospitals, government organizations, military organizations,business enterprises, etc.), entertainment purposes (e.g., for playingvideo games, listening to music, watching video content, etc.), and/orfor accessibility purposes (e.g., as hearing aids, visuals aids, etc.).The embodiments disclosed herein may enable or enhance a user'sartificial reality experience in one or more of these contexts andenvironments and/or in other contexts and environments.

Some AR systems may map a user's environment using techniques referredto as “simultaneous location and mapping” (SLAM). SLAM mapping andlocation identifying techniques may involve a variety of hardware andsoftware tools that can create or update a map of an environment whilesimultaneously keeping track of a user's location within the mappedenvironment. SLAM may use many different types of sensors to create amap and determine a user's position within the map.

SLAM techniques may, for example, implement optical sensors to determinea user's location. Radios including Wi-Fi, Bluetooth, global positioningsystem (GPS), cellular or other communication devices may be also usedto determine a user's location relative to a radio transceiver or groupof transceivers (e.g., a Wi-Fi router or group of GPS satellites).Acoustic sensors such as microphone arrays or 2D or 3D sonar sensors mayalso be used to determine a user's location within an environment. ARand VR devices (such as AR system 100, AR system 200, and VR system 300of FIGS. 1, 2 and 3, respectively) may incorporate any or all of thesetypes of sensors to perform SLAM operations such as creating andcontinually updating maps of the user's current environment. In at leastsome of the embodiments described herein, SLAM data generated by thesesensors may be referred to as “environmental data” and may indicate auser's current environment. This data may be stored in a local or remotedata store (e.g., a cloud data store) and may be provided to a user'sAR/VR device on demand.

FIG. 4 is a block diagram of an example system 400 for transitioningbetween modes of tracking real-world objects for artificial realityinterfaces. As illustrated in this figure, example system 400 mayinclude one or more modules 402 for performing one or more tasks. Aswill be explained in greater detail below, modules 402 may include atracking module 404 that may track (1) a position of a primaryreal-world object within a real-world environment via a primary trackingmethod, and (2) a position of a secondary real-world object within thereal-world environment via a secondary tracking method. Modules 402 mayalso include a presenting module 406 that presents a primary virtualobject that represents the primary real-world object at a positionwithin an artificial environment corresponding to the position of theprimary real-world object within the real-world environment. In someexamples, presenting module 406 may also present a secondary virtualobject that represents the secondary real-world object at a positionwithin the artificial environment corresponding to the position of thesecondary real-world object within the real-world environment.

As further shown in FIG. 400, modules 402 may also include a detectingmodule 408 that detects an interaction of the primary real-world objectwith the secondary real-world object. Modules 402 may further include atransitioning module 410 that transitions from tracking the position ofthe primary real-world object within the real-world environment via theprimary tracking method to tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method in response to detecting the interaction of the primaryreal-world object with the secondary real-world object.

As further illustrated in FIG. 4, example system 400 may also includeone or more memory devices, such as memory 420. Memory 420 generallyrepresents any type or form of volatile or non-volatile storage deviceor medium capable of storing data and/or computer-readable instructions.In one example, memory 420 may store, load, and/or maintain one or moreof modules 402. Examples of memory 420 include, without limitation,Random Access Memory (RAM), Read Only Memory (ROM), flash memory, HardDisk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives,caches, variations or combinations of one or more of the same, or anyother suitable storage memory.

As further illustrated in FIG. 4, example system 400 may also includeone or more physical processors, such as physical processor 430.Physical processor 430 generally represents any type or form ofhardware-implemented processing unit capable of interpreting and/orexecuting computer-readable instructions. In one example, physicalprocessor 430 may access and/or modify one or more of modules 402 storedin memory 420. Additionally or alternatively, physical processor 430 mayexecute one or more of modules 402 to facilitate transitioning betweenmodes of tracking real-world objects for artificial reality interfaces.Examples of physical processor 430 include, without limitation,microprocessors, microcontrollers, central processing units (CPUs),Field-Programmable Gate Arrays (FPGAs) that implement softcoreprocessors, Application-Specific Integrated Circuits (ASICs), portionsof one or more of the same, variations or combinations of one or more ofthe same, or any other suitable physical processor.

As also shown in FIG. 4, example system 400 may also include one or moredata stores, such as data store 440, that may receive, store, and/ormaintain data. Data Store 440 may represent portions of a single datastore or computing device or a plurality of data stores or computingdevices. In some embodiments, data store 440 may be a logical containerfor data and may be implemented in various forms (e.g., a database, afile, a file system, a data structure, etc.). Examples of data store 440may include, without limitation, files, file systems, data stores,databases, and/or database management systems such as an operationaldata store (ODS), a relational database, a NoSQL database, a NewSQLdatabase, and/or any other suitable organized collection of data.

In at least one example, data store 440 may include tracking data 442and/or virtual object data 444. As will be explained in greater detailbelow, in some examples, tracking data 442 may include any informationthat a tracking method may use to identify, calculate, detect, and/orotherwise determine a position of at least one real-world object.Additionally, virtual object data 444 may include any suitable dataassociated with virtual objects that may be presented within anartificial environment including, without limitation, 2D models, 3Dmodels, animation and/or movement data associated with a virtual object,data associated with relationships between and/or among virtual objects,and so forth.

Example system 400 in FIG. 4 may be implemented in a variety of ways.For example, all or a portion of example system 400 may representportions of an example system 500 (“system 500”) in FIG. 5. As shown inFIG. 5, system 500 may include a computing device 502. In at least oneexample, computing device 502 may be programmed with one or more ofmodules 402. Additionally, in some embodiments, system 500 may beassociated with and/or included as part of a suitable artificial realitysystem (e.g., one or more of AR system 100, AR system 200, and/or VRsystem 300).

In at least one embodiment, one or more modules 402 from FIG. 4 may,when executed by computing device 502, enable computing device 502 toperform one or more operations to transition between modes of trackingreal-world objects for artificial reality interfaces. For example, aswill be described in greater detail below, tracking module 404 may causecomputing device 502 to track a position (e.g., primary real-worldposition 504) of a primary real-world object (e.g., primary real-worldobject 506) within a real-world environment (e.g., real-worldenvironment 508) via a primary tracking method (e.g., primary trackingmethod 510). In some examples, tracking module 404 may further causecomputing device 502 to track a position (e.g., secondary real-worldposition 512) of a secondary real-world object (e.g., secondaryreal-world object 514) within the real-world environment via a secondarytracking method (e.g., secondary tracking method 516).

Additionally, presenting module 406 may, when executed by computingdevice 502, cause computing device 502 to present a primary virtualobject (e.g., primary virtual object 518) that represents the primaryreal-world object at a position (e.g., primary artificial environmentposition 520) within an artificial environment (e.g., artificialenvironment 522) corresponding to the position of the primary real-worldobject within the real-world environment. Presenting module 406 mayalso, when executed by computing device 502, cause computing device 502to present a secondary virtual object (e.g., secondary virtual object524) that represents the secondary real-world object at a positionwithin the artificial environment (e.g., secondary artificialenvironment position 526) corresponding to the position of the secondaryreal-world object within the real-world environment.

Furthermore, detecting module 408 may, when executed by computing device502, cause computing device 502 to detect an interaction of the primaryreal-world object with the secondary real-world object (e.g.,interaction 528). Moreover, transitioning module 410 may, when executedby computing device 502, cause computing device 502 to transition (e.g.,indicated in FIG. 5 by transition 530) from tracking the position of theprimary real-world object within the real-world environment via theprimary tracking method to tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method in response to detecting the interaction of the primaryreal-world object with the secondary real-world object.

Computing device 502 generally represents any type or form of computingdevice capable of reading and/or executing computer-executableinstructions. Examples of computing device 502 include, withoutlimitation, servers, desktops, laptops, tablets, cellular phones, (e.g.,smartphones), personal digital assistants (PDAs), multimedia players,embedded systems, wearable devices (e.g., smart watches, smart glasses,etc.), gaming consoles, custom computing devices, combinations of one ormore of the same, or any other suitable mobile computing device.

In at least one example, computing device 502 may be a computing deviceprogrammed with one or more of modules 402. All or a portion of thefunctionality of modules 402 may be performed by computing device 502and/or any other suitable computing system. As will be described ingreater detail below, one or more of modules 402 from FIG. 4 may, whenexecuted by at least one processor of computing device 502 may enablecomputing device 502 to transitioning between modes of trackingreal-world objects for artificial reality interfaces.

Many other devices or subsystems may be connected to example system 400in FIG. 4 and/or example system 500 in FIG. 5. Conversely, all of thecomponents and devices illustrated in FIGS. 4 and 5 need not be presentto practice the embodiments described and/or illustrated herein. Thedevices and subsystems referenced above may also be interconnected indifferent ways from those shown in FIG. 5. Example system 400 andexample system 500 may also employ any number of software, firmware,and/or hardware configurations. For example, one or more of the exampleembodiments disclosed herein may be encoded as a computer program (alsoreferred to as computer software, software applications,computer-readable instructions, and/or computer control logic) on acomputer-readable medium.

FIG. 6 is a flow diagram of an example computer-implemented method 600for allocating shared resources in multi-tenant environments. The stepsshown in FIG. 6 may be performed by any suitable computer-executablecode and/or computing system, including example system 400 in FIG. 4,example system 500 in FIG. 5, and/or variations or combinations of oneor more of the same. In one example, each of the steps shown in FIG. 6may represent an algorithm whose structure includes and/or isrepresented by multiple sub-steps, examples of which will be provided ingreater detail below.

As illustrated in FIG. 6, at step 610, one or more of the systemsdescribed herein may track (1) a position of a primary real-world objectwithin a real-world environment via a primary tracking method, and (2) aposition of a secondary real-world object within the real-worldenvironment via a secondary tracking method. For example, trackingmodule 404 may, as part of computing device 502, cause computing device502 to track primary real-world position 504 of primary real-worldobject 506 within real-world environment 508 via primary tracking method510. Additionally, tracking module 404 may, as part of computing device502, cause computing device 502 to track secondary real-world position512 of secondary real-world object 514 within real-world environment 508via secondary tracking method 516.

In some examples, a “tracking method” may include any suitable methodfor tracking, positioning, locating, and/or orienting a real-worldobject within a real-world environment. By way of example, and withoutlimitation, a tracking method may include an optical tracking methodsuch as a marker-based tracking method, a computer vision trackingmethod, a SLAM tracking method, an inertial tracking method (e.g., atracking method that employs one or more IMUs to track a real-worldobject), combinations or variations of one or more of the same, and soforth.

For example, primary tracking method 510 may include a computer visiontracking method whereby tracking module 404, as part of computing device502, may process data from one or more optical sensors such as 2D or 3Dcameras, time-of-flight depth sensors, single-beam or sweeping laserrangefinders, 3D LiDAR sensors, and/or any other suitable type or formof optical sensors such as may be included in an artificial realitysystem (e.g., VR system 300) to determine and/or track primaryreal-world position 504 of primary real-world object 506.

In some examples, tracking module 404, as part of computing device 502,may process data from the optical sensors in accordance with one or moremachine learning techniques in order to track (e.g., identify, position,locate, and/or orient) primary real-world object 506 within real-worldenvironment 508. For example, in some embodiments, primary real-worldobject 506 may include a hand of a user, and tracking module 404 maygather data from an optical tracking system and/or a marker-basedtracking system to record high-fidelity hand interactions. Trackingmodule 404 may condense the recorded data into 2D imagery and may thenuse the 2D imagery to train one or more convolutional neural networks(CNNs) to identify positions and/or motions of the markers across alarge set of hand pose imagery. This may effectively allow a computingsystem (e.g., computing device 502) to determine a likely position of ahand when later provided with a set of images of a hand of a user.Additionally or alternatively, a suitable CNN may be pre-trained andstored within a suitable data storage location (e.g., as part oftracking data 442 stored within data store 440). Thus, one or more ofmodules 402 (e.g., tracking module 404) may use a suitable sensor (e.g.,an optical sensor included in any of AR system 100, AR system 200, or VRsystem 300) to capture imagery of the user's hands as the user uses anartificial reality system (e.g., any of systems 100, 200, and/or 300),and may use the captured imagery of one or more of the user's hands andthe trained CNN to determine and/or track a position of the user's handor hands within a real-world environment (e.g., real-world environment508).

As another example, secondary tracking method 516 may include an opticaltracking method, a SLAM tracking method, and/or an inertial trackingmethod, and secondary real-world object 514 may include a hand-heldcontroller that may be configured to be tracked via an optical trackingmethod, a SLAM tracking method, and/or an inertial tracking method. Forexample, FIG. 7 illustrates an artificial reality controller device 700(“hand-held controller 700”). Hand-Held controller 700 may be one of apair or set of hand-held controllers associated with an artificialreality system (e.g., one or more of AR system 100, AR system 200,and/or VR system 300). Hand-Held controller 700 includes a main body 702and a handle portion 704 extending from main body 702. In someembodiments, a surrounding ring portion 706 extends from main body 702.As shown in FIG. 700, in some examples, hand-held controller 700 may beconfigured to be operated by a user via a left hand of the user. Inadditional or alternative examples, an artificial reality controllerdevice may be configured to be operated by a user via a right hand ofthe user. In further examples, a hand-held controller may be configuredto be operated by any hand of a user.

As shown in FIG. 7, hand-held controller 700 includes an analog stick708, a trigger button 710, and a third-finger button 712. Main body 702includes a thumb surface 714 from which analog stick 708 extends. Mainbody 702 may also include one or more buttons (e.g., button 716 andbutton 718) positioned on thumb surface 714. In some embodiments, thumbsurface 714 may be a substantially planar surface. Handle portion 704extends from main body 702 on a side generally opposite trigger button710. Third-finger button 712 is operative to detect whether the user isgrasping handle portion 704 with his or her third-finger. In someembodiments, third-finger button 712 may detect various degrees ofdeflection corresponding to the force or pressure of a user's grip onhandle portion 704.

In some embodiments, third-finger button 712 may be active depending onthe context of an associated virtual environment or game. In otherembodiments, third-finger button 712 may be activated mechanically or byanother sensor. In at least one embodiment, handle portion 704 may alsoinclude a palm sensor (e.g., analogous to a pistol grip safety or gripswitch), such that when the palm sensor detects the user's hand, and thethird-finger button 712 is released, an output signal indicates an“open-hand gesture.”

In some embodiments, handle portion 704 may include one or moredetection sensors 720 positioned to detect the presence of the user'spalm or a portion of a finger, indicating that the user is holdinghandle portion 704, indicating how the user is holding handle portion704, and/or how the user is moving his or her hand relative to handleportion 704. For example, detection sensor 720 may include a capacitivetouch sensor on handle portion 704, such as adjacent to third-fingerbutton 712 or in a position for engagement by the user's fourth or fifthfinger when grasping the handle. A detection sensor 720 may bepositioned to be engaged by a portion of the user's second finger (i.e.,index finger) or third finger (i.e., middle finger) that is on handleportion 704 adjacent to trigger button 710 or third-finger button 712,indicating the presence of the user's fingers on handle portion 704 evenif the associated finger has been lifted off of trigger button 710 orthird-finger button 712. Detection sensors 720 may be included on handleportion 704 corresponding to the position of all of the user's fingersgrasping the handle.

In one embodiment, one or more of detection sensors 720 may includeproximity sensors configured to detect a spatial location of a user'sfingers and/or hand relative to handle portion 704. For example,detection sensor 720 could be used to detect a presence of the user'sfinger and/or a separation distance between the respective finger andthe surface of handle portion 704. Detection sensors 720 may beconfigured to allow detection of movement of the user's fingers or otherportions of the user's hand relative to the handle portion 704. Thedetected separation distance and/or movement may be used in connectionwith signals, commands, or other control signals related to the handshape or position of the user's hand or fingers relative to the handleportion 704.

In some embodiments, handle portion 704 may include a combination ofbuttons, pressure sensors, capacitive touch sensors, and/or proximitysensors that may provide signals to initiate a command or to replicate ahand configuration in a corresponding virtual object or avatar.Furthermore, hand-held controller 700 may also include a plurality oftracking features 722 positioned in a corresponding tracking pattern,such as controller tracking pattern 724. Tracking features 722 intracking pattern 724 may be configured to be accurately tracked by asuitable optical tracking system to determine the motion, orientation,and/or spatial position of the controller for reproduction in anartificial environment. Tracking features 722 may include, for example,fiducial markers and/or light emitting diodes (LED).

Furthermore, although not shown in FIG. 7, hand-held controller 700 mayalso include one or more additional electronic elements, one or moreIMUs, one or more additional tracking emitters or detectors, and/or anyother suitable device or system for facilitating tracking of a positionof hand-held controller 700 within a real-world environment viasecondary tracking method 516.

In some examples, primary tracking method 510 and secondary trackingmethod 516 may include different tracking methods. For example, asdescribed above, in at least one embodiment, primary tracking method 510may include a computer vision tracking method and secondary trackingmethod 516 may include an optical tracking method and/or an inertialtracking method. In additional or alternative examples, primary trackingmethod 510 and secondary tracking method 516 may include similartracking methods, although they may remain physically, logically,electrically, optically, communicatively, functionally,methodologically, and/or otherwise distinct from each other. Forexample, primary tracking method 510 and secondary tracking method 516may both include optical tracking methods, but may use differenttechniques, algorithms, and/or devices to track two separate real-worldobjects within a real-world environment.

Furthermore, primary tracking method 510 and secondary tracking method516 may share one or more devices and/or components such as cameras,processors, memory, and so forth while remaining distinct from eachother in other ways. Continuing with the above example, primary trackingmethod 510 and secondary tracking method 516 may both include opticaltracking methods and may use the same image sensor (e.g., camera) tocapture image data. However, each tracking method may use the imagesensor to capture different image data (e.g., different wavelengths,different timings, etc.). Additionally or alternatively, each trackingmethod may access, receive, obtain, etc. image data captured via theimage sensor, but each tracking method may utilize the captured imagedata in different ways (e.g., primary tracking method 510 may processthe captured image data via different techniques and/or algorithms thansecondary tracking method 516).

Tracking module 404 may track (1) primary real-world position 504 ofprimary real-world object 506 within real-world environment 508 viaprimary tracking method 510 and (2) secondary real-world position 512 ofsecondary real-world object 514 within real-world environment 508 viasecondary tracking method 516 in a variety of contexts. For example,primary tracking method 510 and/or secondary tracking method 516 maygenerate and/or receive raw data from one or more sensors included in anartificial reality system (e.g., sensor 240, sensors 220, one or morecamera assemblies, one or more IMUS, one or more proximity sensors,etc.). Tracking module 404 may store this raw data as part of trackingdata 442 within data store 440. Tracking module 404 may additionally oralternatively receive, access, and/or analyze tracking data 442 inaccordance with (1) primary tracking method 510 in order to determineprimary real-world position 504, and (2) in accordance with secondarytracking method 516 in order to determine secondary real-world position512. Tracking module 404 may then store primary real-world position 504and/or secondary real-world position 512 as part of tracking data 442for later use by one or more of modules 402, and/or may transmit primaryreal-world position 504 and/or secondary real-world position 512 to oneor more of modules 402 (e.g., presenting module 406, detecting module408, transitioning module 410, etc.) for use in one or more additionaloperations as described herein.

Hence, one or more of modules 402 (e.g., presenting module 406,detecting module 408, and/or transitioning module 410) may access and/orreceive tracking data 442 to perform one or more operations, such aspresenting a virtual object that corresponds to a tracked real-worldobject within an artificial environment, detecting an interaction ofprimary real-world object 506 with secondary real-world object 514,and/or transitioning from tracking primary real-world object 506 viaprimary tracking method 510 to tracking primary real-world object 506via secondary tracking method 516.

FIG. 8 is a view 800 of an implementation of an example system fortransitioning between modes of tracking real-world objects forartificial reality interfaces. As shown in FIG. 8, a user 804 may wear aVR system 300 within a real-world environment 802. As described above,VR system 300 may include and/or may be in communication with system500, and/or may include modules 402. Furthermore, VR system 300 mayinclude, may implement, and/or may be in communication with primarytracking method 510 and secondary tracking method 516.

As shown, user 804 may be interacting with a virtual environment via VRsystem 300 and may be reaching a hand 806 toward hand-held controller700. Tracking module 404 may be tracking (1) a position of hand 806within real-world environment 802 via primary tracking method 510, and(2) a position of hand-held controller 700 within real-world environment802 via secondary tracking method 516. As will be described in greaterdetail below, upon one or more of modules 402 (e.g., detecting module408) detecting an interaction of hand 806 with hand-held controller 700,one or more of modules 402 (e.g., transitioning module 410) maytransition from tracking hand 806 via primary tracking method 510 totracking hand 806 via secondary tracking method 516, as indicated bytransition indicator 808. Additional examples and explanations oftracking real-world objects via a primary tracking method and asecondary tracking method will be provided in reference to FIGS. 9-13below.

Returning to FIG. 6, at step 620, one or more of the systems describedherein may present (1) a primary virtual object that represents theprimary real-world object at a position within an artificial environmentcorresponding to the position of the primary real-world object withinthe real-world environment, and (2) a secondary virtual object thatrepresents the secondary real-world object at a position within theartificial environment corresponding to the position of the secondaryreal-world object within the real-world environment. For example,presenting module 406 may present primary virtual object 518 at primaryartificial environment position 520 within artificial environment 522corresponding to primary real-world position 504 of primary real-worldobject 506 within real-world environment 508. Additionally, presentingmodule 406 may present secondary virtual object 524 at secondaryartificial environment position 526 within artificial environment 522corresponding to secondary real-world position 512 of secondaryreal-world object 514 within real-world environment 508.

In some examples, an “artificial environment” may include anycomputer-generated environment including, without limitation, anartificial reality environment, a VR environment, an AR environment, a2D environment, a 3D environment, a combination of one or more of thesame, and so forth. In some examples, an artificial environment mayinclude a 2D or 3D representation of a real-world environment (e.g.,real-world environment 508). In some examples, an artificial environment(e.g., artificial environment 522) may be overlaid and/or compositedwith an image of the real-world environment. In some such examples, oneor more virtual objects included in the artificial environment mayappear to a user, when the user views the composite image, to existwithin the real-world environment.

Furthermore, in some examples, a “virtual object” may include, withoutlimitation, any asset, model, object, and/or resource that may bepresented within an artificial environment. In some embodiments, avirtual object may represent an associated real-world object within areal-world environment. For example, a virtual object of a hand of auser may represent a hand of a user within a real-world environment. Asanother example, a virtual object of an artificial reality controllerdevice (e.g., hand-held controller 700) may represent an artificialreality controller device within a real-world environment.

Presenting module 406 may present virtual objects within artificialenvironments in a variety of contexts. For example, presenting module406 may cause a display device included in an artificial reality system(e.g., AR system 200 and/or VR system 300) to display artificialenvironment 522 to a user, and may present one or more virtual objectswithin artificial environment 522.

In some examples, presenting module 406 may present virtual objectswithin artificial environment 522 at positions and/or in configurationsthat may correspond to positions and/or configurations of real-worldobjects within a real-world environment. For example, presenting module406 may present a virtual object at a position within an artificialenvironment by determining a position within an artificial environmentthat corresponds to a position within a real-world environment of acorresponding real-world object.

Presenting module 406 may determine a position within an artificialenvironment (e.g., primary artificial environment position 520 and/orsecondary artificial environment position 526) that corresponds to aposition within a real-world environment (e.g., primary real-worldposition 504 and/or secondary real-world position 512) of acorresponding real-world object (e.g., primary real-world object 506and/or secondary real-world object 514) in any suitable way. Forexample, presenting module 406 may access and/or receive tracking data442 from data store 440 that may include primary real-world position 504and/or secondary real-world position 512. Presenting module 406 may thenidentify a position or positions within artificial environment 522 thatmay correspond to primary real-world position 504 and/or secondaryreal-world position 512 and may designate the identified position and/orpositions as primary artificial environment position 520 and/orsecondary artificial environment position 526.

Furthermore, presenting module 406 may present virtual objects withinartificial environments by identifying an attribute of a real-worldobject and selecting a virtual object to represent the real-world objectbased on the identified attribute. The identified attribute of thereal-world object may include any suitable attribute including, withoutlimitation, a size, an appearance, a shape, a color, a configuration, anorientation, a composition, a position, a relationship to anotherreal-world object, variations or combinations of one or more of thesame, and so forth. In some examples, selecting a virtual object mayinclude adjusting an appearance of the virtual object based on theidentified attribute.

By way of illustration, in at least one embodiment, primary real-worldobject 506 may include a hand of a user. Presenting module 406 mayaccess tracking data 442, which may include an image of primaryreal-world object 506 captured as part of a computer vision trackingmethod. Presenting module 406 may then identify, based on the image, anattribute of primary real-world object 506, such as a shape, size,and/or configuration of primary real-world object 506. Presenting module406 may then select, from virtual object data 444, primary virtualobject 518 based on the attribute. For example, presenting module 406may select a virtual object of a human hand based on primary real-worldobject 506 having a shape of a hand, and may designate the selectedvirtual object as primary virtual object 518. Additionally, presentingmodule 406 may adjust an appearance of the virtual object based on theimage. For example, presenting module 406 may adjust a size of thevirtual object such that an apparent size of primary virtual object 518may appear similar to a size of primary real-world object 506.Presenting module 406 may then present primary virtual object 518 withinartificial environment 522 at primary artificial environment position520.

Hence, presenting module 406 may present a virtual object (e.g., primaryvirtual object 518 and/or secondary virtual object 524) at a position(e.g., a location and/or an orientation) within an artificialenvironment (e.g., primary artificial environment position 520 and/orsecondary artificial environment position 526) such that the positionand/or configuration of the virtual object within the artificialenvironment corresponds to a position and/or configuration of areal-world object (e.g., primary real-world position 504 and/orsecondary real-world position 512) within a real-world environment 508(e.g., real-world environment 508). Additional examples andillustrations of presenting virtual objects within artificialenvironments will be provided below in reference to FIGS. 9-13.

Returning to FIG. 6, at step 630, one or more of the systems describedherein may detect an interaction of a primary real-world object with asecondary real-world object. For example, detecting module 408 may, aspart of computing device 502, detect interaction 528 of primaryreal-world object 506 with secondary real-world object 514.

In some examples, an “interaction” or an “interaction with a real-worldobject” may include any action of a real-world object with respect toanother real-world object. For example, an interaction of primaryreal-world object 506 with secondary real-world object 514 may include,without limitation, a touch of secondary real-world object 514 byprimary real-world object 506, an approach of primary real-world object506 within a predetermined threshold distance of secondary real-worldobject 514, a touching, grasping or lifting of secondary real-worldobject 514 by primary real-world object 506, a release of secondaryreal-world object 514 by primary real-world object 506, a change in aproximity of primary real-world object 506 to secondary real-worldobject 514, and so forth.

Detecting module 408 may detect interaction 528 of primary real-worldobject 506 with secondary real-world object 514 in a variety ofcontexts. For example, in at least one embodiment, detecting module 408may detect interaction 528 of primary real-world object 506 withsecondary real-world object 514 via any suitable combination of controlsand/or sensors included in primary real-world object 506 and/orsecondary real-world object 514, such as via a touch sensor includedprimary real-world object 506 and/or secondary real-world object 514. Insome examples, a “touch sensor” may include any sensor that may that maydetect, capture, and/or record a physical touch of a real-world objectby another real-world object including, without limitation, a physicalobject and/or a human body part (e.g., a hand of a user). In someexamples, a touch sensor may include, without limitation, a capacitivetouch sensor, a resistive touch sensor, an infrared touch sensor, asurface acoustic wave (SAW) touch sensor, a pressure sensor, an inertialsensor (e.g., an IMU), an electric field tomography touch sensor, and soforth. In additional or alternative examples, a touch sensor may detect,capture, and/or record a near proximity of a real-world object toanother real-world object without relying on physical contact.

By way of illustration, as described above, secondary real-world object514 may include an artificial reality controller device that may includea capacitive touch sensor, such as hand-held controller 700. Hence, inat least one embodiment, detecting module 408 may detect interaction 528of primary real-world object 506 with secondary real-world object 514 bydetecting, via a capacitive touch sensor included in secondaryreal-world object 514, a touch of secondary real-world object 514 byprimary real-world object 506 (e.g., a touch of hand-held controller 700by hand 806).

In further embodiments, detecting module 408 may detect interaction 528of primary real-world object 506 with secondary real-world object 514 bydetecting a press of a button included in primary real-world object 506and/or secondary real-world object 514, a change in inertia of primaryreal-world object 506 and/or secondary real-world object 514 via an IMUincluded in primary real-world object 506 and/or secondary real-worldobject 514, and so forth.

In additional or alternative embodiments, detecting module 408 maydetect interaction 528 of primary real-world object 506 with secondaryreal-world object 514 by determining (e.g., via a proximity sensorincluded in at least one of primary real-world object 506 and/orsecondary real-world object 514), a proximity of secondary real-worldobject to primary real-world object 506 (e.g., a proximity of hand 806to hand-held controller 700).

As a simplified example, detecting module 408 may detect an interactionbetween hand 806 and hand-held controller 700 by determining, via aproximity sensor included in a hand-held controller 700 indicating thatan object is within 0.01 m of the hand-held controller, a proximity ofhand 806 to hand-held controller 700 of 0.01 m.

In additional examples, detecting module 408 may determine a proximityof primary real-world object 506 to secondary real-world object 514 bycomparing primary real-world position 504, as tracked by primarytracking method 510, and secondary real-world position 512, as trackedby secondary tracking method 516. Detecting module 408 may thendetermine, based on the comparison, a proximity of primary real-worldobject 506 to secondary real-world object 514.

To illustrate a simplified example, tracking module 404 may track, viaprimary tracking method 510, primary real-world object 506 to a primaryreal-world position 504 with coordinates of [+2 m, +1.5 m, −1.6 m] froma predetermined origin point within real-world environment 508.Likewise, tracking module 404 may track, via secondary tracking method516, secondary real-world object 514 to a secondary real-world position512 with coordinates of [+2 m, +1.7 m, −1.2 m] from the predeterminedorigin point. detecting module 408 may then determine, via a geometricdistance formula, that primary real-world object 506 is 0.45 m fromsecondary real-world object 514.

In some examples, detecting module 408 may detect interaction 528 ofprimary real-world object 506 with secondary real-world object 514 bydetermining that the proximity of primary real-world object 506 tosecondary real-world object 514 is less than a predetermined threshold(e.g., less than 0.0001 m, less than 0.001 m, less than 0.01 m, lessthan 0.1 m, etc.) For example, a predetermined threshold may be 0.05 m.Thus, when detecting module 408 determines that primary real-worldobject 506 is within 0.05 m of secondary real-world object 514,detecting module 408 may detect interaction 528 of primary real-worldobject 506 with secondary real-world object 514.

In some examples, detecting module 408 may combine a variety of trackingdata and/or sensor input to identify primary real-world object 506and/or secondary real-world object 514 and/or to determine a proximityof primary real-world object 506 to secondary real-world object 514.This may, in turn, improve an accuracy and/or robustness of detection ofinteractions between primary real-world object 506 and secondaryreal-world object 514. For example, in at least one embodiment, primarytracking method 510 may include a computer vision tracking method.Tracking module 404 may therefore identify, using one or more computervision techniques, primary real-world object 506 as a hand of a user(e.g., hand 806), and may determine that the hand of the user is at aposition of [+2.5 m, +1.3 m, −2.1 m] from a predetermined origin pointwithin real-world environment 508.

Continuing with this example, secondary tracking method 516 may includean optical tracking method and secondary real-world object 514 mayinclude a hand-held controller 700 configured to be tracked via anoptical tracking method. Tracking module 404 may determine secondaryreal-world position 512 of secondary real-world object 514 of [+2.6 m,+1.4 m, −2.2 m] via secondary tracking method 516. Furthermore, aproximity sensor included in hand-held controller 700 may indicate thatan object is within 0.17 m of hand-held controller 700. Based on thiscombination of information, detecting module 408 may identify hand 806as the object that is within 0.17 m of hand-held controller 700.

Hence, in some embodiments, detecting module 408 may detect aninteraction of a hand of a user (e.g., hand 806) with an artificialreality controller device by identifying primary real-world object 506as a hand of a user and secondary real-world object 514 as an artificialreality controller device. Additionally, detecting module 408 maydetermine a proximity of the hand to the artificial reality controller.This may be more accurate and/or may provide increased capability thansimply determining a proximity of an object to the artificial realitycontroller via a proximity sensor.

It may be noted that some tracking methods may enable one or more ofmodules 402 (e.g., tracking module 404, detecting module 408, etc.) tomake additional determinations regarding one or more real-world objects.For example, a computer vision tracking method may, by analysis of oneor more images of primary real-world object 506 via one or more machinelearning methods, enable detecting module 408 to further determine thatprimary real-world object 506 includes a left hand of a user and/or aright hand of the user.

Furthermore, as described above, in some examples, an artificial realitycontroller device such as hand-held controller 700 may be configured tobe operated by a right hand of a user, a left hand of the user, oreither hand of the user. One or more of modules 402 (e.g., trackingmodule 404, detecting module 408, etc.) may determine whether anartificial reality controller is configured to be operated by a lefthand of a user, a right hand of the user, or either hand of the user.For example, tracking module 404 may determine, via primary trackingmethod 510 (e.g., a computer vision tracking method), secondary trackingmethod 516 (e.g., an optical tracking method configured to track anartificial reality controller device via reference to one or moretracking features included in the artificial reality controller device),that hand-held controller 700 is configured to be operated by a lefthand of a user. Additionally or alternatively, tracking module 404 mayreceive one or more identifiers from hand-held controller 700, via anysuitable communications medium, that may indicate that hand-heldcontroller 700 is configured to be operated by a left hand of the user.

Thus, detecting module 408 may detect an interaction of a hand of a userwith an artificial reality controller device that is configured to beoperated via that hand (e.g., an interaction of a left hand of a userwith a left-handed controller and/or an interaction of a right hand ofthe user with a right-handed controller). Conversely, detecting module408 may detect an interaction of a hand of a user with an artificialreality controller device that is configured to be operated by adifferent hand of the user (e.g., an interaction of a right hand with aleft-handed controller and/or an interaction of a left hand with aright-handed controller).

In some examples, one or more of modules 402 (e.g., presenting module406) may present a notification to a user upon detecting module 408detecting an interaction (e.g., interaction 528) of a hand of the userand an artificial reality controller when the artificial realitycontroller is configured to be operated by the right hand of the userand detecting module 408 determines that the hand is a left hand of theuser. Furthermore, one or more of modules 402 (e.g., presenting module406) may present a notification to the user upon detecting module 408detecting an interaction (e.g., interaction 528) of a hand of the userand an artificial reality controller when the artificial realitycontroller is configured to be operated by the left hand of the user anddetecting module 408 determines that the hand is a right hand of theuser.

By way of illustration, if a user is using an artificial reality systemsuch as illustrated in FIG. 8, hand 806 may include a right hand of user804 and hand-held controller 700 may be configured to be operated by aleft hand of user 804. In such a configuration, detecting module 408 maydetermine that hand 806 is a right hand of user 804, and may detect aninteraction of hand 806 with hand-held controller 700 upon user 804picking up hand-held controller 700 with hand 806. Upon detecting module408 detecting the interaction of hand 806 with hand-held controller 700,presenting module 406 may present a notification (e.g., withinartificial environment 522) to the user that indicates that the user hasattempted to pick up a left-handed controller (e.g., hand-heldcontroller 700) with his or her right hand (e.g., hand 806). Thenotification may also instruct the user to switch hand-held controller700 to the user's left hand. Thus, the systems and methods describedherein may assist users in correct operation of one or more artificialreality systems.

Returning to FIG. 6, at step 640, one or more of the systems describedherein may transition from tracking a position of a primary real-worldobject within a real-world environment via a primary tracking method totracking the position of the primary real-world object within thereal-world environment via the secondary tracking method in response todetecting an interaction of the primary real-world object with thesecondary real-world object. For example, transitioning module 410 may,as part of computing device 502, transition from tracking (e.g., viatracking module 404) primary real-world position 504 of primaryreal-world object 506 within real-world environment 508 via primarytracking method 510 to tracking (e.g., via tracking module 404) primaryreal-world position 504 of primary real-world object 506 withinreal-world environment 508 via secondary tracking method 516.

Transitioning module 410 may transition from tracking (e.g., viatracking module 404) primary real-world position 504 of primaryreal-world object 506 within real-world environment 508 via primarytracking method 510 to tracking (e.g., via tracking module 404) primaryreal-world position 504 of primary real-world object 506 withinreal-world environment 508 via secondary tracking method 516 in avariety of contexts. For example, transitioning module 410 maytransition from tracking primary real-world object 506 via primarytracking method 510 by determining that primary real-world object 506and secondary real-world object 514 are joined together as a unifiedreal-world object, and may track (e.g., cause tracking module 404 totrack) the unified real-world object via secondary tracking method 516.

In additional embodiments, transitioning module 410 may furthertransition from tracking primary real-world object 506 via primarytracking method 510 to tracking primary real-world object 506 withinreal-world environment 508 via secondary tracking method 516 bydeactivating primary tracking method 510. This may enable a computingdevice that implements primary tracking method 510 to free up,deactivate, reallocate, and/or redistribute computing resources (e.g.,processing resources, memory resources, power resources, etc.) thatcomputing device 502 may have previously utilized to execute primarytracking method 510.

By way of illustration, as described above in reference to FIG. 8, insome embodiments, primary real-world object 506 may include a hand of auser such as hand 806, and primary tracking method 510 may include acomputer vision tracking method. Furthermore, in some examples,secondary real-world object 514 may include an artificial realitycontroller, such as hand-held controller 700, and secondary trackingmethod 516 may include an optical tracking method configured to trackhand-held controller 700. One or more of modules 402 (e.g., detectingmodule 408) may detect an interaction of hand 806 with hand-heldcontroller 700, such as user 804 touching, picking up, and/or otherwiseinteracting with hand-held controller 700 via hand 806. In response todetecting the interaction of hand 806 with hand-held controller 700,transitioning module 410 may transition from tracking hand 806 via thecomputer vision tracking method to tracking hand 806 via the opticaltracking method by determining that hand 806 is holding hand-heldcontroller 700, and may therefore cause tracking module 404 to trackhand 806 and hand-held controller 700 as a unified real-world object viathe optical tracking method by tracking hand-held controller 700 via oneor more tracking features 722 included in tracking pattern 724.

Upon transitioning to tracking hand 806 via the optical tracking method,transitioning module 410 may deactivate the computer vision trackingmethod. This may enable computing device 502 to free up, deactivate,reallocate, and/or redistribute computing resources that computingdevice 502 may have used to facilitate the computer vision trackingmethod to other processes, components, and/or devices.

In some embodiments, one or more of modules 402 (e.g., detecting module408) may detect an additional interaction of primary real-world object506 with secondary real-world object 514. In response, one or more ofmodules 402 (e.g., transitioning module 410) may transition fromtracking primary real-world position 504 of primary real-world object506 within real-world environment 508 via secondary tracking method 516to tracking primary real-world position 504 of primary real-world object506 within real-world environment 508 via primary tracking method 510.For example, transitioning module 410 may cause tracking module 404 toreactivate primary tracking method 510 and/or resume tracking primaryreal-world object 506 via primary tracking method 510.

Continuing with the previous illustration, after user 804 picks uphand-held controller 700 with hand 806, user 804 may release hand-heldcontroller 700 and may move hand 806 away from hand-held controller 700.Detecting module 408 may detect the additional interaction of hand 806with controller 700 (e.g., may detect, via a touch sensor included inhand-held controller 700, that user 804 has released hand-heldcontroller 700, and/or may detect an increase in a distance between hand806 and hand-held controller 700) and, in response, transitioning module410 may transition from tracking hand 806 via the optical trackingmethod to tracking hand 806 via the computer vision tracking method. Insome examples, transitioning from tracking hand 806 via the opticaltracking method to tracking hand 806 via the computer vision trackingmethod may include reactivating the computer vision tracking methodand/or resuming tracking hand 806 via the computer vision trackingmethod.

FIGS. 9-13 illustrate various views of virtual objects within anartificial environment in accordance with one or more embodiments of thesystems and methods disclosed herein. FIGS. 9-13 show first-personperspective views of artificial environment 522. In some examples, a VRsystem (e.g., VR system 300) may present artificial environment 522 to auser (e.g., user 804) via a head-worn display system.

Although not shown in FIGS. 9-13, in the scenarios illustrated in FIGS.9-13, one or more of modules 402 (e.g., tracking module 404) may betracking a position of a primary real-world object, such as a positionof a left and/or right hand of user, within a real-world environment viaa primary tracking method (e.g., a computer vision tracking method). Oneor more of modules 402 (e.g., tracking module 404) may also be trackinga position of a secondary real-world object (e.g., a position of aleft-handed and/or a right-handed controller) within the real-worldenvironment via a secondary tracking method (e.g., an optical trackingmethod).

As shown in FIG. 9, view 900 may include a left hand virtual object 902.Left hand virtual object 902 may be a primary virtual object thatrepresents the primary real-world object (e.g., a left hand of a user).Presenting module 406 may be presenting left hand virtual object 902 ata position within artificial environment 522 corresponding to theposition of the primary real-world object within real-world environment508. Furthermore, FIG. 9 also includes a left-handed controller virtualobject 904 that may represent a secondary real-world object (e.g., ahand-held controller 700). Presenting module 406 may be presentingleft-handed controller virtual object 904 at a position withinartificial environment 522 corresponding to the position of thesecondary real-world object within the real-world environment. As willbe described in greater detail below, FIG. 900 further includes a righthand virtual object 906 that may represent a tracked right hand of theuser and a right-handed controller virtual object 908 that may representa tracked right-handed controller. These virtual objects may representadditional real-world objects located in a real-world environment thattracking module 404 may also track in similar ways to how trackingmodule 404 may track the primary and secondary real-world objects.

In some examples, presenting module 406 may adjust an appearance of oneor more virtual objects in order to facilitate user interaction with oneor more corresponding real-world objects. For example, as shown in FIG.9, left-handed controller virtual object 904 and right-handed controllervirtual object 908 may be at least partially visible through left handvirtual object 902 (i.e., left hand virtual object 902 may be at leastpartially transparent with regard to at least left-handed controllervirtual object 904 and right-handed controller virtual object 908),which may aid user 804 in locating hand-held controller 700 withinreal-world environment 802 without removing the head-worn display systemof VR system 300.

Additionally, as described above, in some embodiments, detecting module408 may determine a proximity of a primary real-world object to asecondary real-world object. Therefore, in some examples, presentingmodule 406 may adjust an appearance of one or more virtual objects basedon (e.g., in proportion to) the proximity of the primary real-worldobject to the secondary real-world object.

By way of illustration, FIG. 10 shows an additional view 1000 ofartificial environment 522. In FIG. 10, the user is reaching his or herleft hand, represented by left hand virtual object 902, toward theleft-handed controller, represented by left-handed controller virtualobject 904. As the distance separating the user's left hand and theleft-handed controller decreases, presenting module 406 may adjust, inproportion to the proximity of the user's left hand to the left-handedcontroller, an appearance of left hand virtual object 902 and/orleft-handed controller virtual object 904. For example, and withoutlimitation, presenting module 406 may adjust, in proportion to theproximity of the user's left hand to the left-handed controller, acolor, a size, a relative size, a transparency, a resolution, adefinition, an audio profile, and so forth of left hand virtual object902 and/or left-handed controller virtual object 904. This may provide adynamic feedback mechanism to a user of an artificial reality systemthat may enable the user to locate and/or interact with real-worldobjects within a real-world environment without requiring the user toremove a head-worn display system.

Furthermore, in some examples, presenting module 406 may adjust anappearance of a primary virtual object and/or a secondary virtual objectin response to transitioning module 410 transitioning from tracking theposition of the primary real-world object within the real-worldenvironment via the primary tracking method to tracking the position ofthe primary real world object within the real-world environment via thesecondary tracking method. For example, presenting module 406 mayadjust, without limitation, a color, a size, a relative size, atransparency, a resolution, a definition, an audio profile, and so forthof primary virtual object 518 and/or secondary virtual object 524 inresponse to transitioning module 410 transitioning from tracking (e.g.,via tracking module 404) primary real-world position 504 of primaryreal-world object 506 via primary tracking method 510 to trackingprimary real-world position 504 of primary real-world object 506 viasecondary tracking method 516. This may provide a visual cue to a userof an artificial reality system that may indicate to the user thatprimary real-world object 506 (e.g., a hand of the user) is no longerbeing tracked via primary tracking method 510 and is instead beingtracked via secondary tracking method 516.

In at least some embodiments, presenting module 406 may present aunified virtual object within artificial environment 522 that representsboth primary real-world object 506 and secondary real-world object 514in response to a transition of tracking methods. For example, inresponse to transitioning module 410 transitioning from tracking primaryreal-world position 504 of primary real-world object 506 via primarytracking method 510 to tracking primary real-world position 504 ofprimary real-world object 506 via secondary tracking method 516,presenting module 406 may present a unified virtual object thatrepresents both primary real-world object 506 and secondary real-worldobject 514 at primary artificial environment position 520.

To illustrate, FIG. 11 shows an additional view 1100 of artificialenvironment 522. As shown in FIG. 11, a user's left hand has interactedwith a left-handed controller (e.g., the user has touched, grasped,and/or picked up the left-handed controller with his or her left hand).Detecting module 406 has detected the interaction, and transitioningmodule 410 has transitioned from tracking a position of the user's lefthand via a primary tracking method (e.g., a computer vision trackingmethod) to tracking the position of the user's left hand via a secondarytracking method (e.g., an optical and/or inertial tracking methodconfigured to track a position of the left-handed controller via one ormore tracking features included in the left-handed controller).

In the example illustrated in FIG. 11, presenting module 406 ispresenting a left unified virtual object 1102 at a position withinartificial environment 522 that corresponds to the position of theuser's left hand within the real-world environment. Left unified virtualobject 1102 may represent both the user's left hand that previouslycorresponded to left hand virtual object 902 and the left-handedcontroller that previously corresponded to left-handed controllervirtual object 904. While, as in this example, left unified virtualobject 1102 may replace left hand virtual object 902 and/or left-handedcontroller virtual object 904, in other examples, presenting module 406may continue to present left hand virtual object 902 and/or left-handedcontroller virtual object 904 along with left unified virtual object1102. Additionally or alternatively, presenting module 406 may adjustleft hand virtual object 902 and/or left-handed controller virtualobject 904 in any suitable way upon presenting left unified virtualobject 1102.

In some embodiments, the systems and methods described herein may besimultaneously applied to multiple sets of real-world objects. Forexample, as shown in FIGS. 9-11, tracking module 404 may also track,while tracking a left hand of a user via a primary tracking method(e.g., a computer vision tracking method), a right hand of the user viathe primary tracking method. Likewise, presenting module 406 may alsopresent, while presenting left hand virtual object 902, right handvirtual object 906 at a position within artificial environment 522 thatcorresponds to the tracked position of the right hand of the user withinthe real-world environment. Tracking module 404 and/or presenting module406 may perform these functions in any of the ways described above inreference to a left hand of a user.

Moreover, tracking module 404 may also track, while tracking aleft-handed controller via a secondary tracking method (e.g., an opticaland/or inertial tracking method), a right-handed controller via thesecondary tracking method. Likewise, presenting module 406 may alsopresent, while presenting left-handed controller virtual object 904,right-handed controller virtual object 908 at a position withinartificial environment 522 that corresponds to the tracked position ofthe right-handed controller within the real-world environment. Trackingmodule 404 and/or presenting module 406 may perform these functions inany of the ways described above in reference to a left-handedcontroller.

Furthermore, detecting module 408 may also detect an interaction of aright hand of a user with a right-handed controller in any of the waysdescribed herein, and may similarly transition from tracking the righthand of the user via the primary tracking method to tracking the righthand of the user via the secondary tracking method in any of the waysdescribed herein in reference to the left hand of the user and/or theleft-handed controller.

To illustrate, FIG. 12 includes a view 1200 of artificial environment522. As shown, the user has already picked up the left-handed controllerwith his or her left hand. Detecting module 406 has detected thatinteraction. Transitioning module 410 has also transitioned to trackingthe user's left hand via the secondary tracking method and has replacedleft hand virtual object 902 and left-handed controller virtual object904 with left unified virtual object 1102.

As further shown in FIG. 12, the user is also reaching his or her righthand toward a right-handed controller, as represented respectivelywithin artificial environment 522 by right hand virtual object 906 andright-handed controller virtual object 908. Tracking module 404 may betracking the user's right hand via the primary tracking method (e.g.,the computer vision tracking method) and may be tracking theright-handed controller via the secondary tracking method (e.g., theoptical and/or inertial tracking method). Detecting module 408 maydetect the user picking up the right-handed controller with his or herright hand and may transition from tracking the user's right hand viathe primary tracking method to tracking the user's right hand via thesecondary tracking method.

FIG. 13 includes a view 1300 of artificial environment 522 that includesleft unified virtual object 1102 and right unified virtual object 1302.As shown, transitioning module 410 may have transitioned to tracking theright hand of the user via the secondary tracking method, and presentingmodule 406 may have replaced right hand virtual object 906 andright-handed controller virtual object 908 with a right unified virtualobject 1302. Hence, in this example, one or more of the systemsdescribed herein may be tracking positions of the user's left hand, theuser's right hand, the left-handed controller, and the right-handedcontroller within the real-world environment via the secondary trackingmethod (e.g., via the optical and/or inertial tracking method).

Additionally or alternatively, embodiments of the systems and methodsdescribed herein may perform any other suitable operations in responseto transitioning module 410 transitioning from tracking primaryreal-world object 506 via primary tracking method 510 to trackingprimary real-world object 506 via secondary tracking method 516. By wayof example, one or more of modules 402 (e.g., presenting module 406,detecting module 408, transitioning module 410, etc.) may cause one ormore components of AR system 100, AR system 200, VR system 300, system400, and/or system 500 (e.g., a display device included in AR system 200and/or VR system 300, output audio transducers 306(A) and 306(B), atouch controller, such as hand-held controller 700, etc.) to present anysuitable information (e.g., a visual indication, an audible alert, ahaptic feedback response, etc.) to a user that may indicate that thetransition (e.g., transition 530) has occurred.

As discussed throughout the instant disclosure, the disclosed systemsand methods may provide one or more advantages over traditionalartificial reality systems. For example, embodiments of the systems andmethods described herein may track multiple real-world objects and maypresent, within an artificial environment, virtual objects that mayrepresent the real-world objects and/or may reflect real-world spatialrelationships of those real-world objects.

In one illustration, an embodiment may track a user's hands and anyavailable hand-held controllers that may be within the real-worldenvironment. The embodiment may also present virtual objects thatrepresent the user's hands and any available hand-held controllerswithin an artificial environment. The presented virtual objects may havesimilar spatial relationships within the artificial environment as thereal-world objects have within the real-world environment. Thus, as theuser views the artificial environment via a head-mounted display devicethat may obstruct the user's view of the real-world environment, he orshe may still be able to identify, locate, and/or otherwise interactwith real-world objects (e.g., his or her hands, one or more hand-heldcontrollers, etc.) within the real-world environment.

Additionally, embodiments of the systems and methods described hereinmay enable an artificial reality system to use more efficient, moreaccurate, and/or less resource-intensive tracking methods to trackreal-world objects when those tracking methods may provide equivalent,appropriate and/or otherwise suitable tracking of those real-worldobjects.

For example, while a user is interacting with an artificial realitysystem by using his or her hands (i.e., without a hand-held controller),a computer vision tracking method may accurately track a position of theuser's hands within a real-world environment and may enable accuraterepresentation of the user's hands within an artificial environment.However, while a user interacts with an artificial environment via anoptically trackable hand-held controller, an optical tracking method mayaccurately track a position of the user's hand within the real-worldenvironment by tracking a position of the hand-held controller withinthe real-world environment. Hence, while the user interacts with theartificial environment via the optically trackable hand-held controller,the systems and methods described herein may discontinue the morecomputing-resource-intensive computer vision tracking method, and mayreallocate resources to other operations (e.g., rendering of anartificial environment, processing of audiovisual data, etc.) while theuser is interacting with the artificial environment via the hand-heldcontroller.

Furthermore, the systems and methods described herein may enhanceusability of one or more artificial reality systems. For example, asdescribed above, embodiments of the systems and methods described hereinmay be able to determine whether a user is correctly operating anartificial reality controller configured to be operated by a particularhand of a user (e.g., a left hand or a right hand) with the correcthand. If a user attempts, for example, to pick up a right-handedcontroller with his or her left hand, embodiments of the systems andmethods described herein may present a notification to the user that mayinstruct the user in the proper operation of the artificial realitycontroller device. This may instruct the user in proper operation of theartificial reality system, and/or may prevent the user from operatingthe artificial reality system in an incorrect and/or non-optimal way.

As detailed above, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each include atleast one memory device and at least one physical processor.

Although illustrated as separate elements, the modules described and/orillustrated herein may represent portions of a single module orapplication. In addition, in certain embodiments one or more of thesemodules may represent one or more software applications or programsthat, when executed by a computing device, may cause the computingdevice to perform one or more tasks. For example, one or more of themodules described and/or illustrated herein may represent modules storedand configured to run on one or more of the computing devices or systemsdescribed and/or illustrated herein. One or more of these modules mayalso represent all or portions of one or more special-purpose computersconfigured to perform one or more tasks.

In addition, one or more of the modules described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. For example, one or more of the modules recitedherein may receive tracking data to be transformed, transform thetracking data, output a result of the transformation to track a positionof a primary real-world object within a real-world environment via aprimary tracking method, use the result of the transformation to presenta virtual object at a position within an artificial environment thatcorresponds to the position of the real-world object within thereal-world environment, and store the result of the transformation totransition from tracking the primary real-world object via the primarytracking method to tracking the primary real-world object via asecondary tracking method. Additionally or alternatively, one or more ofthe modules recited herein may transform a processor, volatile memory,non-volatile memory, and/or any other portion of a physical computingdevice from one form to another by executing on the computing device,storing data on the computing device, and/or otherwise interacting withthe computing device.

In some examples, a “computer-readable medium” may include any form ofdevice, carrier, or medium capable of storing or carryingcomputer-readable instructions. Examples of computer-readable mediainclude, without limitation, transmission-type media, such as carrierwaves, and non-transitory-type media, such as magnetic-storage media(e.g., hard disk drives, tape drives, and floppy disks), optical-storagemedia (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), andBLU-RAY disks), electronic-storage media (e.g., solid-state drives andflash media), and other distribution systems.

The process parameters and sequence of the steps described and/orillustrated herein are given by way of example only and can be varied asdesired. For example, while the steps illustrated and/or describedherein may be shown or discussed in a particular order, these steps donot necessarily need to be performed in the order illustrated ordiscussed. The various exemplary methods described and/or illustratedherein may also omit one or more of the steps described or illustratedherein or include additional steps in addition to those disclosed.

The preceding description has been provided to enable others skilled inthe art to best utilize various aspects of the exemplary embodimentsdisclosed herein. This exemplary description is not intended to beexhaustive or to be limited to any precise form disclosed. Manymodifications and variations are possible without departing from thespirit and scope of the instant disclosure. The embodiments disclosedherein should be considered in all respects illustrative and notrestrictive. Reference should be made to the appended claims and theirequivalents in determining the scope of the instant disclosure.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of.” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and have the same meaning as the word“comprising.”

1. A computer-implemented method comprising: tracking: a position of aprimary real-world object within a real-world environment via a primarytracking method; and a position of a secondary real-world object withinthe real-world environment via a secondary tracking method; presenting:a primary virtual object that represents the primary real-world objectat a position within an artificial environment corresponding to theposition of the primary real-world object within the real-worldenvironment; and a secondary virtual object that represents thesecondary real-world object at a position within the artificialenvironment corresponding to the position of the secondary real-worldobject within the real-world environment; detecting: an interaction ofthe primary real-world object with the secondary real-world object; andan additional interaction of the primary real-world object with thesecondary real-world object; determining a proximity of the primaryreal-world object to the secondary real-world object by: determining aposition of the primary real-world object based on the primary trackingmethod; determining a position of the secondary real-world object basedon the secondary tracking method; and comparing the position of theprimary real-world object with the position of the secondary real-worldobject; and transitioning: from tracking the position of the primaryreal-world object within the real-world environment via the primarytracking method to tracking the position of the primary real-worldobject within the real-world environment via the secondary trackingmethod in response to detecting the interaction of the primaryreal-world object with the secondary real-world object; and fromtracking the position of the primary real-world object within thereal-world environment via the secondary tracking method to tracking theposition of the primary real-world object within the real-worldenvironment via the primary tracking method in response to detecting theadditional interaction of the primary real-world object with thesecondary real-world object.
 2. (canceled)
 3. The computer-implementedmethod of claim 1, wherein the primary tracking method comprises acomputer vision tracking method and the secondary tracking methodcomprises at least one of: an optical tracking method; a simultaneouslocalization and mapping (SLAM) tracking method; or an inertial trackingmethod.
 4. The computer-implemented method of claim 1, furthercomprising adjusting, in response to transitioning from tracking theposition of the primary real-world object within the real-worldenvironment via the primary tracking method to tracking the position ofthe primary real-world object within the real-world environment via thesecondary tracking method, an appearance of at least one of: the primaryvirtual object; or the secondary virtual object.
 5. (canceled)
 6. Thecomputer-implemented method of claim 1, wherein detecting theinteraction of the primary real-world object with the secondaryreal-world object comprises determining that the proximity of theprimary real-world object to the secondary real-world object is lessthan a predetermined threshold.
 7. The computer-implemented method ofclaim 1, further comprising adjusting, based on the proximity of theprimary real-world object to the secondary real-world object, anappearance of at least one of: the primary virtual object; or thesecondary virtual object.
 8. The computer-implemented method of claim 1,further comprising presenting, in response to transitioning fromtracking the position of the primary real-world object within thereal-world environment via the primary tracking method to tracking theposition of the primary real-world object within the real-worldenvironment via the secondary tracking method, at the position withinthe artificial environment corresponding to the position of the primaryreal-world object within the real-world environment, a unified virtualobject that represents both the primary real-world object and thesecondary real-world object.
 9. The computer-implemented method of claim1, wherein: the primary real-world object comprises a hand of a user;and the secondary real-world object comprises an artificial realitycontroller device.
 10. The computer-implemented method of claim 1,wherein: the secondary real-world object comprises a touch sensor; anddetecting the interaction of the primary real-world object with thesecondary real-world object comprises detecting, via the touch sensor, atouch of the secondary real-world object by the primary real-worldobject.
 11. The computer-implemented method of claim 1, furthercomprising determining that the primary real-world object comprises oneof: a left hand of a user; or a right hand of the user.
 12. Thecomputer-implemented method of claim 11, wherein: the secondaryreal-world object comprises an artificial reality controller deviceconfigured to be operated by the user via one of: the left hand of theuser; or the right hand of the user; and the computer-implemented methodfurther comprises presenting a notification to the user upon detectingthe interaction of the primary real-world object with the secondaryreal-world object and when at least one of: the artificial realitycontroller device is configured to be operated by the right hand of theuser and upon determining that the primary real-world object comprisesthe left hand of the user; or the artificial reality controller deviceis configured to be operated by the left hand of the user and upondetermining that the primary real-world object comprises the right handof the user.
 13. A system comprising: a tracking module, stored inmemory, that tracks: a position of a primary real-world object within areal-world environment via a primary tracking method; and a position ofa secondary real-world object within the real-world environment via asecondary tracking method; a presenting module, stored in memory, thatpresents: a primary virtual object that represents the primaryreal-world object at a position within an artificial environmentcorresponding to the position of the primary real-world object withinthe real-world environment; and a secondary virtual object thatrepresents the secondary real-world object at a position within theartificial environment corresponding to the position of the secondaryreal-world object within the real-world environment; a detecting module,stored in memory, that detects: an interaction of the primary real-worldobject with the secondary real-world object; an additional interactionof the primary real-world object with the secondary real-world object;and a proximity of the primary real-world object to the secondaryreal-world object by: determining a position of the primary real-worldobject based on the primary tracking method; determining a position ofthe secondary real-world object based on the secondary tracking method;and comparing the position of the primary real-world object with theposition of the secondary real-world object; a transitioning module,stored in memory, that transitions: from tracking the position of theprimary real-world object within the real-world environment via theprimary tracking method to tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method in response to detecting the interaction of the primaryreal-world object with the secondary real-world object; and fromtracking the position of the primary real-world object within thereal-world environment via the secondary tracking method to tracking theposition of the primary real-world object within the real-worldenvironment via the primary tracking method in response to detecting theadditional interaction of the primary real-world object with thesecondary real-world object; and at least one physical processor thatexecutes the tracking module, the presenting module, the detectingmodule, and the transitioning module.
 14. (canceled)
 15. The system ofclaim 13, wherein the presenting module further adjusts, in response tothe transitioning module transitioning from tracking the position of theprimary real-world object within the real-world environment via theprimary tracking method to tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method, an appearance of at least one of: the primary virtualobject; or the secondary virtual object.
 16. (canceled)
 17. The systemof claim 13, wherein the detecting module detects the interaction of theprimary real-world object with the secondary real-world object bydetermining that the proximity of the primary real-world object to thesecondary real-world object is less than a predetermined threshold. 18.The system of claim 13, wherein the presenting module adjusts, based onthe proximity of the primary real-world object to the secondaryreal-world object, an appearance of at least one of: the primary virtualobject; or the secondary virtual object.
 19. The system of claim 13,wherein: the secondary real-world object comprises an artificial realitycontroller device configured to be operated by a user via one of: a lefthand of the user; or a right hand of the user; the detecting modulefurther determines that the primary real-world object comprises one of:the left hand of the user; or the right hand of the user; and thepresenting module further presents a notification to the user upon thedetecting module detecting the interaction of the primary real-worldobject with the secondary real-world object and when at least one of:the artificial reality controller device is configured to be operated bythe right hand of the user and upon the detecting module determiningthat the primary real-world object comprises the left hand of the user;or the artificial reality controller device is configured to be operatedby the left hand of the user and upon the detecting module determiningthat the primary real-world object comprises the right hand of the user.20. A non-transitory computer-readable medium comprising instructionsthat, when executed by at least one processor of a computing system,cause the computing system to: track: a position of a primary real-worldobject within a real-world environment via a primary tracking method;and a position of a secondary real-world object within the real-worldenvironment via a secondary tracking method; present: a primary virtualobject that represents the primary real-world object at a positionwithin an artificial environment corresponding to the position of theprimary real-world object within the real-world environment; and asecondary virtual object that represents the secondary real-world objectat a position within the artificial environment corresponding to theposition of the secondary real-world object within the real-worldenvironment; detect: an interaction of the primary real-world objectwith the secondary real-world object; and an additional interaction ofthe primary real-world object with the secondary real-world object;determine a proximity of the primary real-world object to the secondaryreal-world object by: determining a position of the primary real-worldobject based on the primary tracking method; determining a position ofthe secondary real-world object based on the secondary tracking method;and comparing the position of the primary real-world object with theposition of the secondary real-world object; and transition: fromtracking the position of the primary real-world object within thereal-world environment via the primary tracking method to tracking theposition of the primary real-world object within the real-worldenvironment via the secondary tracking method in response to detectingthe interaction of the primary real-world object with the secondaryreal-world object; and from tracking the position of the primaryreal-world object within the real-world environment via the secondarytracking method to tracking the position of the primary real-worldobject within the real-world environment via the primary tracking methodin response to detecting the additional interaction of the primaryreal-world object with the secondary real-world object.
 21. Thecomputer-implemented method of claim 1, wherein the secondary virtualobject is at least partially visible through the primary virtual objectwithin the artificial environment.
 22. (canceled)